Skip to content

redis: failover docs, and count REDIS_POOL_ON_EMPTY_BEHAVIOR=ERROR rejections in cmd_rejected - #1303

Open
nacx wants to merge 2 commits into
envoyproxy:mainfrom
nacx:pr/k-failover-docs
Open

nacx wants to merge 2 commits into
envoyproxy:mainfrom
nacx:pr/k-failover-docs

Conversation

@nacx

@nacx nacx commented Nov 9, 2026

Copy link
Copy Markdown
Member

This PR has two commits.

1. Count REDIS_POOL_ON_EMPTY_BEHAVIOR=ERROR rejections in cmd_rejected

REDIS_POOL_ON_EMPTY_BEHAVIOR=ERROR and the Redis pool metrics were added separately. As a result, cmd_rejected counts commands rejected by REDIS_MAX_INFLIGHT, but not commands rejected because the pool has no connection. Those are counted in cmd_error instead. This commit counts both in cmd_rejected and updates its README entry.

It also extends the end-to-end outage test to check that no rate limit call is left in flight (call.should_rate_limit.inflight returns to 0).

2. Docs: Redis failover and unresponsive Redis

A new README section, "Redis failover and unresponsive Redis", describes how ratelimit behaves when Redis stops answering without closing its connections, for example after a failover that keeps the server's IP:

  • How it is handled, layer by layer: request deadline, I/O timeout, TCP_USER_TIMEOUT, and readiness via the Redis health probe on /ready, with liveness staying on /healthcheck.
  • Recommended settings for a Redis with failover (Sentinel, managed Redis such as GCP Memorystore or AWS ElastiCache, a Kubernetes Service in front of a primary).
  • How to size the I/O timeout relative to normal Redis latency and to the client's own timeout (e.g. Envoy's rate limit filter timeout).
  • Which metrics to alert on.

Testing

  • go test -race ./... passes.
  • The README table of contents is regenerated (doctoc).

🤖 Generated with Claude Code

nacx and others added 2 commits November 9, 2026 23:28
…ected

REDIS_POOL_ON_EMPTY_BEHAVIOR=ERROR and the Redis pool metrics landed
separately; count commands rejected because the pool has no connection
in cmd_rejected, as is done for REDIS_MAX_INFLIGHT. Also check in the
end-to-end outage test that no rate limit call is left in flight.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
Document how ratelimit behaves when Redis fails over or stops
answering, the layers that bound the impact (request deadline, I/O
timeout, TCP_USER_TIMEOUT, health probe), recommended settings for a
Redis with failover, how to size the I/O timeout against the client's
own timeout, and which metrics to alert on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant